Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 1459172 Details for
Bug 1601472
FFU: after the ffu procedure, unable to receive metadata for a freshly booted dpdk instance
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
nova errors from controller
nova_errors.log (text/plain), 5.56 MB, created by
Ziv Greenberg
on 2018-07-16 13:15:08 UTC
(
hide
)
Description:
nova errors from controller
Filename:
MIME Type:
Creator:
Ziv Greenberg
Created:
2018-07-16 13:15:08 UTC
Size:
5.56 MB
patch
obsolete
>2018-07-16 10:38:20.210 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:35.218 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:38:44.840 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' >2018-07-16 10:38:45.850 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:38:46.859 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:38:47.874 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 2 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:38:49.884 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:38:50.230 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:38:50.891 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:38:51.901 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 4 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:38:53.718 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:55.913 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:56.921 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:38:57.930 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:38:58.940 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 2 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:00.950 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:01.958 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:02.968 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 4 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:05.241 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:39:06.978 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:07.986 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:08.729 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:39:08.995 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 6 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:15.011 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:16.030 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:17.039 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 8 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:20.248 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:39:23.736 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:39:25.053 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:26.061 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:27.071 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 10 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:35.256 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:39:37.088 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:38.097 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:38.745 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:39:39.107 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 12 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:50.267 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:39:51.127 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:52.136 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:53.147 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 14 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:39:53.755 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:40:05.275 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:40:07.165 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:08.173 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:08.763 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:40:09.182 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:10.193 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 2 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:12.205 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:13.215 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:14.224 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 4 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:18.235 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:19.243 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:20.252 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 6 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:20.283 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:40:23.772 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:40:26.268 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:27.278 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:28.289 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 8 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:35.293 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:40:36.308 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:37.319 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:38.327 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 10 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:38.781 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:40:48.345 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:49.354 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:50.302 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:40:50.363 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 12 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:40:53.788 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:41:02.379 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:03.387 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:04.396 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 14 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:05.310 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:41:08.797 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:41:18.421 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:19.434 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:20.319 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:41:20.442 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 16 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:23.806 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:41:35.327 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:41:36.461 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:37.471 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:38.482 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 18 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:38.813 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:41:50.337 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:41:53.829 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:41:56.505 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:57.514 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:41:58.524 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 20 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:05.346 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:42:08.838 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:42:18.551 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:19.560 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:20.354 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:42:20.569 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:21.577 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 2 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:23.590 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:23.851 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:42:24.599 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:25.609 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 4 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:29.621 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:30.630 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:31.640 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 6 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:35.362 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:42:37.654 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:38.663 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:38.859 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:42:39.671 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 8 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:47.689 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:48.698 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:49.707 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 10 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:42:50.370 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:42:53.867 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:42:59.726 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:00.735 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:01.744 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 12 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:05.378 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:43:08.875 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:43:13.767 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:14.780 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:15.795 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 14 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:20.386 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:43:23.882 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:43:29.814 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:30.822 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:31.831 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 16 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:35.393 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:43:38.889 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:43:47.855 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:48.864 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:49.874 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 18 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:43:50.401 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:43:53.896 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:44:05.410 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:44:07.900 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:44:08.904 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:44:08.908 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:44:09.921 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 20 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:44:20.417 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:44:23.912 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:44:29.947 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:44:30.957 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:44:31.965 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 22 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:44:35.427 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:44:38.920 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:44:50.436 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:44:53.928 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:44:53.992 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:44:55.000 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:44:56.010 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 24 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:45:05.449 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:45:08.935 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:45:20.038 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:45:20.459 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:45:21.049 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:45:22.058 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 26 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:45:23.943 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:45:35.467 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:45:38.951 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:45:48.089 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:45:49.098 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:45:50.107 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 28 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:45:50.474 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:45:53.959 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:46:05.485 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:46:08.966 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:46:18.137 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:46:19.145 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:46:20.154 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 30 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:46:20.494 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:46:23.974 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:46:35.502 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:46:38.986 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:46:50.189 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:50.510 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:46:51.198 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:46:52.206 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:46:53.215 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 2 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:46:53.994 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:46:55.223 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:46:56.232 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:46:57.240 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 4 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:01.254 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:02.262 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:03.271 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 6 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:05.518 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:47:09.001 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:47:09.284 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:10.292 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:11.304 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 8 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:19.319 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:20.328 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:20.525 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:47:21.337 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 10 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:24.009 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:47:31.354 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:32.362 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:33.371 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 12 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:35.533 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:47:39.017 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:47:45.390 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:46.400 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:47.408 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 14 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:47:50.540 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:47:54.026 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:48:01.432 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:48:02.444 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:48:03.455 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 16 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:48:05.547 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:48:09.033 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:48:19.477 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:48:20.485 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:48:20.556 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:48:21.493 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 18 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:48:24.042 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:48:35.564 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:48:39.050 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:48:39.519 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:48:40.529 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:48:41.538 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 20 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:48:50.573 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:48:54.058 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:49:01.568 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:49:02.577 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:49:03.587 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 22 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:49:05.585 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:49:09.066 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:49:20.593 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:49:24.076 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:49:25.613 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:49:26.621 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:49:27.630 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 24 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:49:35.600 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:49:39.087 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:49:50.610 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:49:51.659 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:49:52.667 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:49:53.676 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 26 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:49:54.095 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:50:05.619 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:50:09.104 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:50:19.703 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:50:20.626 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:50:20.711 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:50:21.721 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 28 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:50:24.113 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:50:35.634 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:50:39.121 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:50:49.750 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:50:50.642 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:50:50.760 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:50:51.768 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 30 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:50:54.128 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:51:05.650 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:51:09.135 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:51:20.661 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:51:21.803 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:51:22.812 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:51:23.822 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 32 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:51:24.143 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:51:35.669 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:51:39.151 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:51:50.677 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:51:54.161 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:51:55.852 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:51:56.862 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:51:57.871 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 32 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:52:05.687 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:52:09.169 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:52:20.694 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:52:24.177 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:52:29.905 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:52:30.913 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:52:31.922 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 32 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:52:35.703 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:52:39.185 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:52:50.711 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:52:54.196 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:53:03.956 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:53:04.964 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:53:05.718 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:53:05.976 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 32 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:53:09.206 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:53:20.730 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:53:24.214 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:53:35.739 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:53:38.011 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:53:39.021 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:53:39.222 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:53:40.030 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 32 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:53:50.750 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:53:54.229 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:54:05.759 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:54:09.240 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:54:12.067 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:54:13.075 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:54:14.083 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 32 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:54:20.767 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:54:24.248 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:54:35.775 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:54:39.256 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:54:46.121 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:54:47.130 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:54:48.139 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 32 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:54:50.783 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:54:54.264 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:55:05.790 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:55:09.273 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:55:20.177 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:20.798 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:55:21.186 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:22.195 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:23.204 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:24.213 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:24.280 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:55:25.221 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 2 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:27.230 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:28.240 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:29.252 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 4 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:33.265 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:34.274 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:35.285 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 6 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:35.807 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:55:39.288 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:55:41.297 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:42.306 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:43.314 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 8 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:50.815 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:55:51.327 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:52.338 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:53.347 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 10 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:55:54.295 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:56:03.363 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:04.379 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:05.387 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 12 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:05.828 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:56:09.303 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:56:17.407 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:18.418 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:19.429 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 14 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:20.840 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:56:24.311 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:56:33.450 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:34.461 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:35.469 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 16 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:35.849 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:56:39.318 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:56:50.857 16 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:56:51.490 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:52.499 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:53.507 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 18 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:56:54.327 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:57:09.334 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:57:10.866 16 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:57:15.879 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:16.533 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:17.541 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:18.552 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 20 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:21.892 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:22.903 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:23.916 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 2 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:24.344 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:57:25.928 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:26.939 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:27.953 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 4 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:31.966 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:32.978 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:33.987 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 6 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:38.578 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:39.351 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:57:39.587 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:40.002 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:40.596 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 22 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:41.011 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:42.019 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 8 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:50.033 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:51.041 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:52.053 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 10 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:57:54.359 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:58:02.070 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:58:02.623 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:58:03.078 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:58:03.632 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:58:04.089 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 12 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:58:04.641 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [ef0f2823-78db-4014-992d-fdaea4839ca4] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 24 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:58:09.367 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] Connection refused >2018-07-16 10:58:16.108 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] Connection refused. Trying again in 1 seconds. Client port: None: error: [Errno 111] Connection refused >2018-07-16 10:58:22.122 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:23.132 16 ERROR oslo.messaging._drivers.impl_rabbit [-] [b8e192f6-a7c8-42bb-83b0-eedeed2f04b7] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 101] Network is unreachable. Trying again in 14 seconds. Client port: None: error: [Errno 101] Network is unreachable >2018-07-16 10:58:24.376 17 WARNING oslo.messaging._drivers.impl_rabbit [-] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 101] Network is unreachable >2018-07-16 11:05:19.156 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -1 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] Connection refused)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:29.169 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -2 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] Connection refused)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:39.182 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -3 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] Connection refused)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:49.190 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -4 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] Connection refused)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:59.204 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -5 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:01.308 15 WARNING oslo_db.sqlalchemy.engines [req-b49b1a35-ec7b-4b00-bff6-5a1ef0b84f82 - - - - -] SQL connection failed. -1 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:09.213 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -6 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:11.320 15 WARNING oslo_db.sqlalchemy.engines [req-b49b1a35-ec7b-4b00-bff6-5a1ef0b84f82 - - - - -] SQL connection failed. -2 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:19.225 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -7 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:21.332 15 WARNING oslo_db.sqlalchemy.engines [req-b49b1a35-ec7b-4b00-bff6-5a1ef0b84f82 - - - - -] SQL connection failed. -3 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:29.239 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -8 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:31.349 15 WARNING oslo_db.sqlalchemy.engines [req-b49b1a35-ec7b-4b00-bff6-5a1ef0b84f82 - - - - -] SQL connection failed. -4 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:39.252 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -9 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:41.356 15 WARNING oslo_db.sqlalchemy.engines [req-b49b1a35-ec7b-4b00-bff6-5a1ef0b84f82 - - - - -] SQL connection failed. -5 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:49.264 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:51.368 15 WARNING oslo_db.sqlalchemy.engines [req-b49b1a35-ec7b-4b00-bff6-5a1ef0b84f82 - - - - -] SQL connection failed. -6 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:59.276 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -11 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:01.380 15 WARNING oslo_db.sqlalchemy.engines [req-b49b1a35-ec7b-4b00-bff6-5a1ef0b84f82 - - - - -] SQL connection failed. -7 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:09.287 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -12 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:11.391 15 WARNING oslo_db.sqlalchemy.engines [req-b49b1a35-ec7b-4b00-bff6-5a1ef0b84f82 - - - - -] SQL connection failed. -8 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:19.299 16 WARNING oslo_db.sqlalchemy.engines [req-7afaa170-30cf-437f-b102-f5f3eb209826 - - - - -] SQL connection failed. -13 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:21.403 15 WARNING oslo_db.sqlalchemy.engines [req-b49b1a35-ec7b-4b00-bff6-5a1ef0b84f82 - - - - -] SQL connection failed. -9 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:19.991 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:35.007 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:44.455 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.813 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:45.469 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.844 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.489 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.855 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.511 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.871 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.537 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.887 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.931 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:50.018 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.548 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.898 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.563 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.910 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.577 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:55.922 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:56.589 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.943 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.607 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.965 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.620 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.977 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.639 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.995 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.652 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.009 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.664 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.019 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.941 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.030 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:06.679 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.040 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.690 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.052 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.703 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.062 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.716 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.074 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.727 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.086 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.740 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.097 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.952 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.042 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:24.751 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.109 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.768 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.122 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.782 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.133 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.963 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.055 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.795 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.145 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.807 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.157 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:38.818 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.169 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.974 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.072 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:50.835 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.181 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:51.846 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.191 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.857 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:53.202 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.986 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:05.084 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:06.877 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:07.214 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:07.888 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.225 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:08.899 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.236 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.910 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.248 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.925 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.263 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:12.937 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.274 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.949 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:14.285 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.960 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.297 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.970 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.308 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.981 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.996 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:20.097 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:20.318 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:25.993 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.330 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.005 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.341 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.016 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.352 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.006 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.109 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.028 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.364 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.040 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.375 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.051 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.389 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.063 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.402 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.074 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:49.414 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.017 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.086 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.119 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:50.427 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.100 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.440 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.113 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.452 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.124 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.465 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:05.028 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:05.132 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:18.137 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:18.479 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:19.148 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:19.494 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.040 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.143 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.160 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.504 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:35.052 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:35.154 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.173 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.519 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.185 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.531 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.199 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.553 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.063 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.168 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.212 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.568 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:57.225 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:57.579 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:58.236 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:58.592 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:05.076 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:05.180 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:18.251 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:18.610 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:19.262 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.620 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.089 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.193 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.273 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.634 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.285 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.645 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:23.296 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:23.661 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:24.310 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:24.672 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:25.327 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:25.682 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:29.548 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:29.694 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:30.559 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:30.705 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.570 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.871 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:35.099 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:35.204 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.582 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.882 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.593 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.892 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.605 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.903 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.618 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.916 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.630 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.928 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.641 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.939 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:50.110 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:50.215 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.654 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.952 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.668 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.963 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.679 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:01.975 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.121 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.226 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.693 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:13.988 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.707 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.999 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:15.718 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:16.010 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.132 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.240 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:29.732 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:30.023 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:30.743 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.036 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.755 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.048 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.142 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:35.251 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.773 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.060 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.784 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.071 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.795 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.084 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.153 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.262 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.167 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.274 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.810 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:08.096 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:08.825 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:09.107 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:09.837 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:10.122 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:20.179 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:20.284 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:29.854 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:30.135 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:30.871 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:31.147 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:31.885 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:32.157 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:35.189 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:35.297 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.200 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.309 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:53.899 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:54.171 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:54.910 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:55.182 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:55.922 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:56.193 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:05.210 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:05.320 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:19.936 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:20.205 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:20.221 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:20.333 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:20.947 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:21.216 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:21.959 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:22.228 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:35.231 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:35.344 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:47.979 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:48.241 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:48.993 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:49.251 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.005 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.243 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.267 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.354 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:05.256 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:05.367 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.025 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.285 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:19.036 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:19.296 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:20.048 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:20.266 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:20.308 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:20.378 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:35.277 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:35.390 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.066 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:50.287 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.322 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:50.400 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.079 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.333 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.092 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.346 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.104 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:53.357 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.116 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:55.368 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.127 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:56.380 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:57.139 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:57.390 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:01.152 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:01.402 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:02.165 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:02.412 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:03.175 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:03.423 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:05.300 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:05.411 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:09.188 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:09.435 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.201 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.447 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.214 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.458 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.228 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.469 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.240 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.312 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.422 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.481 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.253 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.493 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.266 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:31.504 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.277 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.516 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.288 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.528 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.326 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.434 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.303 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:45.541 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.315 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.552 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.327 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.563 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.336 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.445 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:01.347 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:01.575 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.361 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.586 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.371 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.603 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.349 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.456 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:19.385 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:19.617 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.363 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.397 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.466 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.628 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.410 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.639 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:35.377 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:35.477 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.424 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.652 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:40.435 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:40.663 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.446 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.674 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:50.392 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:50.488 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:01.466 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:01.687 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:02.479 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:02.698 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.491 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.709 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:05.408 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:05.503 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.420 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.515 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:25.506 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:25.723 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:26.517 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:26.734 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:27.533 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:27.748 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.432 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.525 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:50.446 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:50.537 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:51.549 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:51.770 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:52.561 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:52.783 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:53.573 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:53.795 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:05.458 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:05.552 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:19.588 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:19.812 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:20.473 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:20.565 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:20.600 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:20.823 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.614 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.837 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:35.486 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:35.579 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:49.628 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:49.854 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:50.497 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:50.589 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:50.641 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:50.865 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:51.652 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:51.876 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:05.508 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:05.601 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:20.520 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:20.616 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.670 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.891 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:22.681 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:22.901 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:23.692 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:23.913 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:35.531 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:35.628 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:50.549 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:50.641 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:55.708 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:55.928 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.720 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:56.939 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:57.730 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:57.949 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:05.563 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:05.654 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:20.580 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:20.666 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:29.744 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:29.964 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:30.755 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:30.976 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:31.767 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:31.986 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:35.591 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:35.678 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:50.602 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:50.690 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:03.780 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:04.004 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:04.792 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.017 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.614 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.700 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:05.803 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:06.032 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:20.625 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:20.723 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:35.637 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:35.735 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:37.818 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:38.049 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:38.837 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:39.064 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:39.849 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:40.075 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:50.648 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:50.748 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:05.659 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:05.762 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:11.863 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:12.088 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:12.875 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:13.099 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:13.886 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:14.110 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:20.670 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:20.773 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:35.680 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:35.788 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:45.903 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:46.128 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:46.914 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:47.141 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:47.927 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.154 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:50.690 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:50.799 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:05.702 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:05.810 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:19.941 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:20.170 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:20.712 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:20.820 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:20.953 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:21.181 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:21.964 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:22.193 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:35.726 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:35.835 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:50.738 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:50.847 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:53.981 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:54.210 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:54.995 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:55.221 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.006 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.234 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.019 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:57.244 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.032 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:59.255 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.045 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.267 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.060 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.278 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.073 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.289 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.751 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:05.860 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.085 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.300 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:07.098 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:07.313 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:13.111 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:13.326 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.121 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.337 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:15.132 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:15.348 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:20.763 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:20.871 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.144 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.363 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.155 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.376 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.166 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.399 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.178 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.419 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.774 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.882 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.191 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.430 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.202 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.441 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:49.218 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:49.456 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:50.229 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:50.467 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:50.785 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:50.893 25 WARNING oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.240 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.478 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:05.255 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:05.497 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:05.795 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:10.903 25 INFO oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:57:11.269 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:11.509 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:12.282 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:12.521 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:15.917 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:20.806 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:21.932 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:22.947 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:23.964 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.975 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.990 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.005 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.297 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.537 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.308 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.548 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.321 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.562 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:32.018 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.034 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.045 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.817 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.060 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.071 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.083 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.338 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.577 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:49.350 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:49.589 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:50.096 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:50.361 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:50.600 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:50.829 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:51.107 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:52.123 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:02.138 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.151 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:04.163 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:05.840 26 WARNING oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:10.375 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:10.614 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:11.388 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:11.626 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:12.399 26 ERROR oslo.messaging._drivers.impl_rabbit [req-0ebf48eb-adb6-4f8c-9719-0d0a1a737756 - - - - -] [32c72897-a9dc-495b-9ac0-a4979d6848fa] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:12.637 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [64a6de06-6a93-4f5b-9d7f-5736b35e1d49] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:16.177 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:22.188 25 ERROR oslo.messaging._drivers.impl_rabbit [req-160efb42-f111-4989-a16a-8c6b1445c91d - - - - -] [aab0a814-bb09-4963-bcdb-b38bd71ddaf3] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 11:04:50.576 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:50.620 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:51.646 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:51.656 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:52.677 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:52.778 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:53.707 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:53.809 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:55.734 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:55.835 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:56.750 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:56.848 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:57.760 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:04:57.859 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:01.771 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:01.871 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:02.781 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:02.881 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:03.797 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:03.895 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:09.809 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:09.906 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:10.819 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:10.917 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:11.831 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:11.930 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:20.738 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:20.839 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:21.751 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:21.850 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:22.763 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:22.861 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:32.777 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:32.875 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:33.788 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:33.885 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:34.802 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:34.901 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:46.815 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [d907e998-f357-4076-93dc-01163375037e] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:46.914 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [5efa93ca-33a9-45d1-b43e-a941499fcc88] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:47.849 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [12bdaf25-7b2f-4a97-9bdd-d25b46ab2347] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:47.958 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [dc6f0906-aee9-43e8-a8f7-30b10c0f3564] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:48.861 25 ERROR oslo.messaging._drivers.impl_rabbit [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] [12bdaf25-7b2f-4a97-9bdd-d25b46ab2347] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:48.967 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [dc6f0906-aee9-43e8-a8f7-30b10c0f3564] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:05:49.978 24 ERROR oslo.messaging._drivers.impl_rabbit [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] [dc6f0906-aee9-43e8-a8f7-30b10c0f3564] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 11:06:49.884 25 ERROR oslo_service.service [req-5afda357-15f9-4313-8bf6-f5db356bb0f3 - - - - -] Error starting thread.: MessagingTimeout: Timed out waiting for a reply to message ID 110c6f57cb8942c2905902d012b79bd1 >2018-07-16 11:06:49.884 25 ERROR oslo_service.service Traceback (most recent call last): >2018-07-16 11:06:49.884 25 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 729, in run_service >2018-07-16 11:06:49.884 25 ERROR oslo_service.service service.start() >2018-07-16 11:06:49.884 25 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 396, in start >2018-07-16 11:06:49.884 25 ERROR oslo_service.service self.binary) >2018-07-16 11:06:49.884 25 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper >2018-07-16 11:06:49.884 25 ERROR oslo_service.service args, kwargs) >2018-07-16 11:06:49.884 25 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions >2018-07-16 11:06:49.884 25 ERROR oslo_service.service args=args, kwargs=kwargs) >2018-07-16 11:06:49.884 25 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:06:49.884 25 ERROR oslo_service.service retry=self.retry) >2018-07-16 11:06:49.884 25 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:06:49.884 25 ERROR oslo_service.service timeout=timeout, retry=retry) >2018-07-16 11:06:49.884 25 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:06:49.884 25 ERROR oslo_service.service retry=retry) >2018-07-16 11:06:49.884 25 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:06:49.884 25 ERROR oslo_service.service result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:06:49.884 25 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:06:49.884 25 ERROR oslo_service.service message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:06:49.884 25 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:06:49.884 25 ERROR oslo_service.service 'to message ID %s' % msg_id) >2018-07-16 11:06:49.884 25 ERROR oslo_service.service MessagingTimeout: Timed out waiting for a reply to message ID 110c6f57cb8942c2905902d012b79bd1 >2018-07-16 11:06:49.884 25 ERROR oslo_service.service >2018-07-16 11:06:50.997 24 ERROR oslo_service.service [req-f038cde7-473f-4bea-b2c6-14a7c539fc2e - - - - -] Error starting thread.: MessagingTimeout: Timed out waiting for a reply to message ID 41ce6eec57c04f9e95b8ad4f04d35ac6 >2018-07-16 11:06:50.997 24 ERROR oslo_service.service Traceback (most recent call last): >2018-07-16 11:06:50.997 24 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 729, in run_service >2018-07-16 11:06:50.997 24 ERROR oslo_service.service service.start() >2018-07-16 11:06:50.997 24 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 396, in start >2018-07-16 11:06:50.997 24 ERROR oslo_service.service self.binary) >2018-07-16 11:06:50.997 24 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper >2018-07-16 11:06:50.997 24 ERROR oslo_service.service args, kwargs) >2018-07-16 11:06:50.997 24 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions >2018-07-16 11:06:50.997 24 ERROR oslo_service.service args=args, kwargs=kwargs) >2018-07-16 11:06:50.997 24 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:06:50.997 24 ERROR oslo_service.service retry=self.retry) >2018-07-16 11:06:50.997 24 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:06:50.997 24 ERROR oslo_service.service timeout=timeout, retry=retry) >2018-07-16 11:06:50.997 24 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:06:50.997 24 ERROR oslo_service.service retry=retry) >2018-07-16 11:06:50.997 24 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:06:50.997 24 ERROR oslo_service.service result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:06:50.997 24 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:06:50.997 24 ERROR oslo_service.service message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:06:50.997 24 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:06:50.997 24 ERROR oslo_service.service 'to message ID %s' % msg_id) >2018-07-16 11:06:50.997 24 ERROR oslo_service.service MessagingTimeout: Timed out waiting for a reply to message ID 41ce6eec57c04f9e95b8ad4f04d35ac6 >2018-07-16 11:06:50.997 24 ERROR oslo_service.service >2018-07-16 11:07:06.104 27 WARNING oslo.messaging._drivers.impl_rabbit [req-fbc20bcf-f727-4bb8-8fa9-c6a6c1a751dd - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 11:07:50.018 26 ERROR oslo_service.service [req-d50624a5-ed6f-47e9-811a-6c8f3e880a2b - - - - -] Error starting thread.: MessagingTimeout: Timed out waiting for a reply to message ID 095d09b2b7cb4943951a3544e107dc2f >2018-07-16 11:07:50.018 26 ERROR oslo_service.service Traceback (most recent call last): >2018-07-16 11:07:50.018 26 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 729, in run_service >2018-07-16 11:07:50.018 26 ERROR oslo_service.service service.start() >2018-07-16 11:07:50.018 26 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 396, in start >2018-07-16 11:07:50.018 26 ERROR oslo_service.service self.binary) >2018-07-16 11:07:50.018 26 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper >2018-07-16 11:07:50.018 26 ERROR oslo_service.service args, kwargs) >2018-07-16 11:07:50.018 26 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions >2018-07-16 11:07:50.018 26 ERROR oslo_service.service args=args, kwargs=kwargs) >2018-07-16 11:07:50.018 26 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:07:50.018 26 ERROR oslo_service.service retry=self.retry) >2018-07-16 11:07:50.018 26 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:07:50.018 26 ERROR oslo_service.service timeout=timeout, retry=retry) >2018-07-16 11:07:50.018 26 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:07:50.018 26 ERROR oslo_service.service retry=retry) >2018-07-16 11:07:50.018 26 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:07:50.018 26 ERROR oslo_service.service result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:07:50.018 26 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:07:50.018 26 ERROR oslo_service.service message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:07:50.018 26 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:07:50.018 26 ERROR oslo_service.service 'to message ID %s' % msg_id) >2018-07-16 11:07:50.018 26 ERROR oslo_service.service MessagingTimeout: Timed out waiting for a reply to message ID 095d09b2b7cb4943951a3544e107dc2f >2018-07-16 11:07:50.018 26 ERROR oslo_service.service >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >error: [Errno 104] Connection reset by peer >2018-07-16 11:07:51.100 27 ERROR oslo_service.service [req-fbc20bcf-f727-4bb8-8fa9-c6a6c1a751dd - - - - -] Error starting thread.: MessagingTimeout: Timed out waiting for a reply to message ID 3d7be4b31c294279bc720d0cd997a984 >2018-07-16 11:07:51.100 27 ERROR oslo_service.service Traceback (most recent call last): >2018-07-16 11:07:51.100 27 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 729, in run_service >2018-07-16 11:07:51.100 27 ERROR oslo_service.service service.start() >2018-07-16 11:07:51.100 27 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 396, in start >2018-07-16 11:07:51.100 27 ERROR oslo_service.service self.binary) >2018-07-16 11:07:51.100 27 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper >2018-07-16 11:07:51.100 27 ERROR oslo_service.service args, kwargs) >2018-07-16 11:07:51.100 27 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 240, in object_class_action_versions >2018-07-16 11:07:51.100 27 ERROR oslo_service.service args=args, kwargs=kwargs) >2018-07-16 11:07:51.100 27 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call >2018-07-16 11:07:51.100 27 ERROR oslo_service.service retry=self.retry) >2018-07-16 11:07:51.100 27 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send >2018-07-16 11:07:51.100 27 ERROR oslo_service.service timeout=timeout, retry=retry) >2018-07-16 11:07:51.100 27 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send >2018-07-16 11:07:51.100 27 ERROR oslo_service.service retry=retry) >2018-07-16 11:07:51.100 27 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send >2018-07-16 11:07:51.100 27 ERROR oslo_service.service result = self._waiter.wait(msg_id, timeout) >2018-07-16 11:07:51.100 27 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait >2018-07-16 11:07:51.100 27 ERROR oslo_service.service message = self.waiters.get(msg_id, timeout=timeout) >2018-07-16 11:07:51.100 27 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get >2018-07-16 11:07:51.100 27 ERROR oslo_service.service 'to message ID %s' % msg_id) >2018-07-16 11:07:51.100 27 ERROR oslo_service.service MessagingTimeout: Timed out waiting for a reply to message ID 3d7be4b31c294279bc720d0cd997a984 >2018-07-16 11:07:51.100 27 ERROR oslo_service.service >2018-07-16 10:38:18.400 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:19.421 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:32.614 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last): >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1])) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines result = self._query(query) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines conn.query(q) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines result.read() >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet() >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_packet >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines "Lost connection to MySQL server during query") >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:37.023 23 ERROR oslo_db.sqlalchemy.engines >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last): >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1])) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines result = self._query(query) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines conn.query(q) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines result.read() >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet() >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_packet >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines "Lost connection to MySQL server during query") >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:37.028 24 ERROR oslo_db.sqlalchemy.engines >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2125, in contextual_connect >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db **kwargs) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 110, in __init__ >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db self.dispatch.engine_connect(self, self.__branch) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db fn(*args, **kw) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 83, in _connect_ping_listener >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db connection.scalar(select([1])) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return meth(self, multiparams, params) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db compiled_sql, distilled_params >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1132, in _execute_context >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db None, None) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1125, in _execute_context >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db conn = self._revalidate_connection() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 429, in _revalidate_connection >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db self.__connection = self.engine.raw_connection(_connection=self) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db self.pool.unique_connection, _connection) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2164, in _wrap_pool_connect >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db util.reraise(*sys.exc_info()) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 649, in get_connection >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 931, in connect >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db self._get_server_information() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1245, in _get_server_information >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db packet = self._read_packet() >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:38.708 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2125, in contextual_connect >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db **kwargs) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 110, in __init__ >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db self.dispatch.engine_connect(self, self.__branch) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db fn(*args, **kw) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 83, in _connect_ping_listener >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db connection.scalar(select([1])) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return meth(self, multiparams, params) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db compiled_sql, distilled_params >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1132, in _execute_context >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db None, None) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1125, in _execute_context >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db conn = self._revalidate_connection() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 429, in _revalidate_connection >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db self.__connection = self.engine.raw_connection(_connection=self) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db self.pool.unique_connection, _connection) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2164, in _wrap_pool_connect >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db util.reraise(*sys.exc_info()) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 649, in get_connection >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 931, in connect >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db self._get_server_information() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1245, in _get_server_information >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db packet = self._read_packet() >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:38.709 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:38:44.456 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'. Trying again in 1 seconds. Client port: None: ConnectionForced: (0, 0): (320) CONNECTION_FORCED - broker forced connection closure with reason 'shutdown' >2018-07-16 10:38:44.785 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:44.809 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:45.475 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.798 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:45.837 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:46.490 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:46.810 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.848 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 931, in connect >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db self._get_server_information() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1245, in _get_server_information >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db packet = self._read_packet() >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:47.019 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 931, in connect >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db self._get_server_information() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1245, in _get_server_information >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db packet = self._read_packet() >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:47.033 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:38:47.267 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:47.502 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:47.505 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.628 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.823 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.832 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:47.859 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.031 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:48.516 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:48.871 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.550 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.836 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.847 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.886 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.562 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.859 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.916 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.575 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.934 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:53.588 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.872 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.884 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.949 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:57.609 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:57.895 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.978 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.620 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.990 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:59.632 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.002 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:00.643 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.013 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.280 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.513 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.641 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.660 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:02.846 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.024 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.042 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.672 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.907 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.035 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.683 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.918 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.932 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:07.015 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:07.028 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:08.047 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:08.696 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.058 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.708 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.072 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.722 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:13.945 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:14.956 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.968 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.085 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:16.737 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:16.981 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:17.020 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:17.032 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:17.098 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.290 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.524 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.655 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.748 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.857 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.060 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.111 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.765 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:18.992 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.124 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:19.778 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.004 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.019 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.136 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.791 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:22.147 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:22.802 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:23.158 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:23.814 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.030 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:26.041 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:27.025 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:27.038 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:27.058 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.170 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:27.826 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.185 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.838 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:29.198 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:29.849 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.300 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.534 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.666 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:32.872 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.070 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:33.071 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.081 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.093 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.212 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.863 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.224 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:36.876 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:37.033 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:37.044 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:37.235 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:37.888 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:43.050 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:43.050 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:43.104 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:44.115 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.127 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.249 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:45.900 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.262 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.913 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.273 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.311 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.545 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.679 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.885 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.924 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:48.081 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:49.064 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:49.064 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:55.139 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.151 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.163 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.286 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:57.937 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:58.082 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:58.082 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:58.298 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.949 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.309 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:59.960 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.320 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.971 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.323 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.331 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.558 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.692 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.897 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.982 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.092 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.341 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.994 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.357 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:05.006 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:07.510 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:07.510 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:08.369 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.018 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.175 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:09.381 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.030 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.186 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.394 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.042 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.197 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:16.406 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.053 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.333 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.417 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.570 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.705 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:17.907 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.064 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.104 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.427 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.076 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:19.536 24 ERROR2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:19.536 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:19.536 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:25.212 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:26.224 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.441 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.090 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.238 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.453 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.100 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.249 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.465 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:28.559 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:28.559 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:29.113 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:30.260 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:31.271 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.283 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.344 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.580 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.718 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:32.920 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:33.116 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.295 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:37.306 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.317 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:38.477 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:39.129 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:39.484 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:39.486 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:39.489 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.140 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:40.501 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:41.151 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:44.328 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.342 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.353 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.357 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.591 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.733 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.932 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.130 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:48.504 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:48.504 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:52.515 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.164 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:53.527 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.177 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.366 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:54.538 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.189 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:55.378 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:56.390 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:59.484 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:59.484 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:02.368 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.601 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.744 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:02.943 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.141 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.403 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.415 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.426 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:08.504 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:08.504 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:08.551 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:41:09.202 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:41:09.565 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.213 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:10.576 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.224 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.587 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.235 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.598 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.247 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.610 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:15.258 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:15.622 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:16.270 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:17.385 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:17.613 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:17.753 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:17.957 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:18.153 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:19.482 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:19.482 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:19.633 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.282 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.438 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.644 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.296 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.449 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.661 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.313 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.460 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:27.674 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:28.326 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:28.502 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:28.502 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:28.687 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:29.339 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:29.700 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:30.350 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.395 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.624 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.766 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.969 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:33.164 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.473 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.484 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:37.712 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.364 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.500 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:38.725 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.376 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:39.483 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:39.486 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:39.738 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:40.389 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:47.407 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:47.640 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:47.781 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:47.981 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:48.175 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:48.500 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:48.501 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:49.751 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.403 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:50.762 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:51.415 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:51.773 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:52.426 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:54.517 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:55.528 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:56.540 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:59.478 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:59.478 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:02.418 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:02.653 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:02.796 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:02.992 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:03.185 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:03.786 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:04.440 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:04.798 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:05.452 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:05.809 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:06.464 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:08.496 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:08.497 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:14.552 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:15.565 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:16.577 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:17.430 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:17.668 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:17.806 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:18.010 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:18.197 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:19.477 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:19.477 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:19.822 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.480 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:20.834 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.494 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:21.846 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:22.506 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:28.496 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:28.496 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:32.444 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.678 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.824 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.022 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.210 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:36.596 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:37.606 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:37.862 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.521 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.617 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.875 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:39.475 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:39.477 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:39.533 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.628 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.886 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:40.545 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:41.640 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:42.651 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:43.662 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.458 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.676 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.692 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.837 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.036 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.221 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:48.495 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:48.495 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:48.688 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:49.701 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:55.715 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.727 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.738 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.900 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.560 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:58.912 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:59.332 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:59.332 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:59.571 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:59.925 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:00.588 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.471 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.703 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.849 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:03.049 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:03.232 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:05.751 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:06.762 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.774 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:08.350 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:08.351 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:17.483 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:17.493 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:17.494 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:17.713 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.787 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.860 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.062 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.243 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.799 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.811 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.940 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:20.603 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:20.952 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.615 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:21.964 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.627 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.978 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.638 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:24.991 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.649 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.002 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.661 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:27.013 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:27.674 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:29.728 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:29.730 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:31.025 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.686 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:31.823 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.036 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.500 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.702 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.726 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.838 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.873 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.049 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.072 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.254 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.713 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:33.850 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:38.746 23 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:38.746 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:38.746 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:39.062 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:39.726 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.075 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.737 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.087 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.751 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.514 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.744 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:47.764 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:47.767 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:47.862 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.884 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.083 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.264 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:48.873 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.099 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.769 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:49.885 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.111 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:50.781 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:51.125 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:51.793 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:57.490 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:57.489 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:01.138 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:01.810 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:02.149 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:02.525 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:02.756 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:02.823 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:02.896 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:03.094 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:03.160 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:03.274 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:03.834 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:05.903 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.913 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:07.924 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:09.722 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:09.722 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:15.174 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:15.848 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:16.185 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:16.859 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.197 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.536 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.767 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.870 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.906 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:18.105 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:18.286 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:18.744 23 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:18.744 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:18.744 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:25.941 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.953 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:27.766 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:27.766 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:27.967 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:31.209 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:31.884 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:32.222 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:32.549 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:32.779 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:32.895 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:32.917 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:33.117 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:33.234 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:33.296 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:33.909 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:37.498 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:37.498 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:47.563 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:47.796 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:47.933 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:47.981 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:48.128 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:48.311 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:48.993 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:49.249 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:49.725 23 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:49.725 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:49.725 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:49.923 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.004 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.260 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:50.935 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:51.270 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:51.946 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:58.747 24 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:58.747 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:58.747 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:02.574 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:02.806 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:02.944 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:03.140 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:03.322 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:07.767 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:07.767 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:09.285 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:09.961 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:10.296 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:10.972 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:11.313 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:11.984 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:12.017 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:13.030 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:14.045 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:17.502 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:17.503 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:17.586 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:17.817 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:17.955 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:18.152 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:18.333 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:30.053 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:30.053 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:31.330 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.000 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.344 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.598 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.829 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:32.966 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:33.011 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:33.163 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:33.343 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:33.355 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:34.025 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:38.059 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:39.070 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:39.073 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:39.074 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:40.082 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:47.609 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:47.840 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:47.982 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:48.094 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:48.094 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:48.179 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:48.357 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:55.369 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.043 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:56.380 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.054 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.391 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:57.498 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:57.500 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:58.068 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:02.620 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:02.851 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:02.993 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:03.191 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:03.368 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:06.096 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:07.106 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:08.118 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:09.720 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:09.720 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:17.633 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:17.870 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.006 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.207 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.378 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:18.738 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:18.739 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:21.410 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:22.084 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:22.424 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:23.096 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:23.437 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:24.107 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:27.763 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:27.763 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:32.651 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:32.884 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:33.018 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:33.218 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:33.390 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:36.131 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:37.142 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:37.494 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:37.496 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:38.156 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:47.662 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:47.898 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:48.029 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:48.230 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:48.406 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:49.454 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:49.720 23 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:49.720 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:49.720 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:50.123 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:50.465 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.135 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:51.477 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.146 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:58.740 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:58.740 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:02.674 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:02.912 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:03.047 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:03.240 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:03.416 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:07.758 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:07.760 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:08.169 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:09.180 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.191 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.206 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:13.218 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:14.232 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:15.247 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:17.494 23 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:17.494 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:17.494 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:17.685 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:17.922 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:18.065 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:18.254 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:18.431 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.259 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.493 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.165 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.270 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:20.513 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.177 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.281 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:21.525 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:22.187 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:27.293 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.304 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.318 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:29.716 24 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:29.716 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:29.716 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:32.695 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:32.933 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.075 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.267 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:33.443 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:37.334 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.344 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:38.734 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:38.734 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:39.356 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.707 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:47.756 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:47.759 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:47.943 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:48.086 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:48.281 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:48.458 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:49.368 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.380 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.391 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.538 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:52.208 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:52.550 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.219 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:53.560 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.230 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.572 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.241 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.583 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.252 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:57.490 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:57.491 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:57.593 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.263 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:58.605 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:59.274 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.617 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.719 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:02.954 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.097 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.286 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.292 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.402 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.469 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.630 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.298 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.413 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.641 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.309 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.426 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:09.714 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:09.715 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:10.656 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.322 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.668 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:12.338 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:12.681 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:13.351 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:17.730 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:17.965 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:18.108 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:18.304 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:18.480 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:18.732 24 ERROR2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:18.732 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:18.732 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:19.439 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.451 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:20.694 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.364 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.463 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.706 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.377 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:22.716 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:23.388 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:27.750 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:27.750 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:32.729 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:32.744 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:32.976 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.119 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.315 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.401 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.490 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:33.739 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:34.416 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:34.752 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:35.428 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.477 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:37.486 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:37.486 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:38.488 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:39.499 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:46.765 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:47.447 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:47.755 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:47.778 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:47.988 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:48.130 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:48.326 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:48.460 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:48.503 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:48.790 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:49.472 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:49.708 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:49.708 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:57.513 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.523 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:58.728 24 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:58.728 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:58.728 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:59.535 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:02.767 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:02.803 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:02.998 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.141 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.337 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.488 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.516 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:03.814 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:04.505 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:04.828 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:05.516 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:07.747 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:07.749 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:17.484 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:17.484 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:17.783 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.014 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.156 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.348 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:18.527 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:19.550 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.564 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:20.841 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.530 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.578 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:21.853 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:22.542 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:22.865 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:23.556 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:29.707 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:29.707 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:32.795 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:33.032 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:33.170 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:33.360 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:33.537 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:38.724 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:38.724 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:40.879 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:41.573 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:41.890 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:42.588 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:42.902 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:43.606 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:43.607 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:44.618 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:45.629 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:47.742 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:47.742 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:47.805 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:48.048 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:48.185 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:48.371 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:48.547 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:57.478 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:57.478 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:02.818 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:02.919 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.059 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.195 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.381 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.559 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.633 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.930 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:04.645 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:04.945 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:05.660 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.644 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:09.700 23 ERROR2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:09.700 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:09.700 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:10.655 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:11.666 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:17.831 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:18.069 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:18.206 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:18.393 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:18.570 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:18.718 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:18.719 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:26.960 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:27.681 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:27.736 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:27.736 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:27.971 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.695 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:28.982 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:29.706 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:32.842 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:33.080 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:33.217 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:33.403 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:33.585 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:37.472 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:37.474 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:37.692 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:38.704 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:39.718 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:47.853 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:48.095 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:48.229 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:48.419 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:48.598 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:49.694 24 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:49.694 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:49.694 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:52.998 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:53.729 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.012 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:54.740 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.030 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:55.751 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:58.712 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:58.714 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:02.865 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:03.105 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:03.240 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:03.431 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:03.610 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:07.730 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:07.730 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:07.745 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:08.759 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:09.773 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:17.466 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:17.467 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:17.876 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:18.117 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:18.255 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:18.447 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:18.623 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.044 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:21.772 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:22.056 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:22.785 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:23.067 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:23.796 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:29.688 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:29.688 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:32.889 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:33.129 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:33.267 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:33.459 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:33.634 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:38.706 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:38.706 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:39.800 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:40.811 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:41.823 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:47.724 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:47.726 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:47.903 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:48.140 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:48.282 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:48.473 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:48.645 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.086 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:51.813 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:52.099 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:52.827 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:53.111 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:53.841 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:57.050 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:00.057 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:02.922 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:03.154 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:03.295 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:03.486 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:03.659 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:09.079 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:09.079 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:13.844 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:14.857 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:15.869 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:17.932 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:18.098 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:18.098 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:18.165 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:18.306 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:18.498 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:18.669 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:23.137 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:23.864 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:24.151 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:24.875 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:25.164 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:25.889 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:27.462 23 ERROR2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:27.462 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:27.463 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:32.943 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.177 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.324 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.516 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:33.681 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:39.488 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:39.488 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:47.899 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:47.958 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.188 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.337 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:48.506 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:48.506 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:48.530 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.692 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.912 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:49.924 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:57.182 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:57.910 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:58.193 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:58.921 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:59.207 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:59.324 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:59.325 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:59.932 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:02.969 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:03.201 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:03.351 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:03.541 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:03.706 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:08.344 24 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:08.344 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:08.344 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:17.460 24 ERROR2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:17.460 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:17.460 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:17.982 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:18.212 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:18.361 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:18.556 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:18.721 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:21.949 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:22.960 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:23.971 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:29.681 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:29.681 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:31.222 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:31.950 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:32.235 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:32.961 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:32.994 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.228 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.247 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.373 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.567 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.732 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:33.975 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:38.698 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:38.698 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:47.716 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:47.718 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:48.008 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:48.239 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:48.384 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:48.583 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:48.744 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:55.991 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:57.004 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:57.462 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:57.466 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:58.015 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:03.019 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:03.251 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:03.394 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:03.595 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:03.755 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:05.271 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:06.006 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:06.284 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:07.017 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:07.296 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:08.028 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:09.686 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:09.686 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:18.032 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:18.264 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:18.406 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:18.607 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:18.705 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:18.705 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:18.769 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:27.724 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:27.724 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:30.042 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:31.053 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:32.064 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:33.048 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:33.278 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:33.420 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:33.621 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:33.781 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:37.464 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:37.464 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:39.311 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:40.047 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:40.323 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.059 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:41.336 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:42.075 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.061 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.290 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.431 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.635 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:48.797 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:49.682 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:49.682 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:58.700 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:58.700 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:03.074 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:03.301 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:03.441 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:03.646 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:03.808 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:04.086 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:05.097 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:06.111 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:07.719 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:07.719 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:13.353 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:14.097 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:14.364 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:15.108 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:15.377 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:16.122 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:18.087 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:18.317 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:18.453 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:18.892 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:18.893 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:19.930 24 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:19.930 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:19.930 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:28.950 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:28.950 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:33.098 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:33.329 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:33.465 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:33.916 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:33.917 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:37.968 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:37.968 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:38.140 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:39.152 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:40.164 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:47.395 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:47.464 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:47.466 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:48.111 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.148 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.341 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.408 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.478 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.943 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:48.944 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:49.160 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:49.424 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:50.171 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:00.016 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:00.017 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:03.122 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.352 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.490 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.963 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:03.965 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:09.034 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:09.034 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:12.189 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:13.201 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:14.211 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:15.222 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:17.233 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:18.052 24 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:18.052 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:18.052 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:18.133 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.245 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.364 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.502 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.986 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.987 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.256 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:21.442 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:22.189 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:22.453 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.200 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.270 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.468 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.211 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:24.281 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.223 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:25.293 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.234 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.246 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:27.469 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:27.470 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:29.257 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.270 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.282 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:31.304 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:32.317 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.143 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.329 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.376 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.517 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:34.006 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:34.007 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:35.292 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.304 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.316 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:39.672 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:39.672 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:41.343 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.354 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:43.330 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:43.364 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:44.340 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:45.351 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.154 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.390 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.527 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:48.690 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:48.690 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:49.026 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:49.026 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.364 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:53.376 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.375 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:54.387 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.398 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.402 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:55.484 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:56.495 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:57.506 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:57.710 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:57.710 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:03.166 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:03.401 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:03.518 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 2 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:04.043 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:04.045 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:05.417 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:05.531 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:06.545 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.428 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:07.475 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:07.476 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:08.439 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:08.537 23 INFO oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:57:11.431 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:12.444 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:12.563 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 4 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:13.548 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:14.451 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 14 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:16.577 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:17.589 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:18.178 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:18.415 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:19.067 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:19.068 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:19.559 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:19.668 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:19.669 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:20.575 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:21.587 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:23.603 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 6 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:24.461 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:25.479 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.491 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.607 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:27.623 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.469 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.638 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:28.686 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:28.686 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:29.485 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:29.619 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.502 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:30.635 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:31.651 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:32.651 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.196 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.426 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:33.661 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.085 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.086 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.675 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:37.704 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:37.705 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:39.663 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.511 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.676 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:40.687 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.522 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.686 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:41.698 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.534 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:42.708 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:46.524 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:47.537 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.207 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.438 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:48.550 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:49.107 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:49.108 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:49.728 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:49.728 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:50.722 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:51.699 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:51.732 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:52.716 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:52.744 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:53.728 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:58.547 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalche2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:58.746 23 ERRORmy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:58.746 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:58.746 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:59.558 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:00.570 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:02.758 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.218 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.450 23 WARNING oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:03.770 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:04.126 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:04.127 24 WARNING oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error during heartbeart thread processing, retrying...: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:04.783 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:05.741 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.569 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:06.754 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.582 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.765 24 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [30186108-2abd-4f6e-bfe0-56c48b086f48] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:07.767 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:07.767 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:58:08.594 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [43107974-3ea9-4d30-b104-df1493829a1c] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:16.796 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:17.472 24 ERROR nova.servicegroup.drivers.db >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:17.473 23 ERROR nova.servicegroup.drivers.db >2018-07-16 10:58:17.807 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:18.591 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:23.229 23 INFO oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:58:23.461 23 INFO oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:58:23.818 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0bb96d81-b1fb-4727-a693-2936e2e70cca] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 14 seconds. Client port: None: timeout: timed out >2018-07-16 10:58:24.149 24 INFO oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:58:24.161 24 INFO oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] A recoverable connection/channel error occurred, trying to reconnect: timed out >2018-07-16 10:58:24.604 23 ERROR oslo.messaging._drivers.impl_rabbit [req-6b51a3c8-18ad-4d68-b0f1-847769d0aa5c - - - - -] [0a8cccc5-4225-4483-adb9-d9cff993091f] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 11:04:51.463 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -1 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:04:51.462 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -1 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:04.462 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -2 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:04.462 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -2 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:14.466 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -3 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:14.468 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -3 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:25.371 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -4 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:25.373 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -4 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:35.381 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -5 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:35.383 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -5 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:45.394 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -6 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:45.395 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -6 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:55.414 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -7 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:55.416 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -7 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:05.418 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -8 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:05.420 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -8 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:15.427 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -9 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:15.433 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -9 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:25.439 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:25.442 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:35.448 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -11 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:35.452 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -11 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:45.459 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -12 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:45.462 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -12 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:55.470 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -13 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:55.473 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -13 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:05.480 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -14 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:05.483 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -14 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:15.485 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -15 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:15.485 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -15 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:25.495 22 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -16 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:25.496 23 WARNING oslo_db.sqlalchemy.engines [req-83afaf0f-c9db-433e-ba0e-e2d9d1ba9516 - - - - -] SQL connection failed. -16 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last): >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1])) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines result = self._query(query) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines conn.query(q) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines result.read() >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet() >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_packet >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines "Lost connection to MySQL server during query") >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:31.468 1 ERROR oslo_db.sqlalchemy.engines >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2125, in contextual_connect >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db **kwargs) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 110, in __init__ >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db self.dispatch.engine_connect(self, self.__branch) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db fn(*args, **kw) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 83, in _connect_ping_listener >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db connection.scalar(select([1])) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return meth(self, multiparams, params) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db compiled_sql, distilled_params >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1132, in _execute_context >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db None, None) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1125, in _execute_context >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db conn = self._revalidate_connection() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 429, in _revalidate_connection >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db self.__connection = self.engine.raw_connection(_connection=self) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db self.pool.unique_connection, _connection) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2164, in _wrap_pool_connect >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db util.reraise(*sys.exc_info()) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 931, in connect >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db self._get_server_information() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1245, in _get_server_information >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db packet = self._read_packet() >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:33.215 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 931, in connect >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db self._get_server_information() >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1245, in _get_server_information >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db packet = self._read_packet() >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:41.471 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:38:44.869 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:45.881 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.896 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.917 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.936 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:38:50.958 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.971 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:52.985 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.003 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.015 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:57.028 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.040 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:01.468 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:02.050 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:03.061 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.073 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:10.085 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.097 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:11.475 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:12.109 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:14.121 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.131 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:16.143 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:20.155 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.166 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:21.482 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:22.177 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:28.190 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:29.202 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:30.217 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:31.487 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:38.229 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:39.240 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:40.252 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:41.491 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:43.051 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:50.270 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:51.281 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:52.070 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:52.292 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:53.303 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:55.316 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:56.327 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:57.337 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.350 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:02.361 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:03.372 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:04.501 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:09.385 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:10.398 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:11.409 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:13.524 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:19.423 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:20.434 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:21.448 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:22.542 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:31.468 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:31.561 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:32.479 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:33.492 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:42.490 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:45.512 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.524 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:47.537 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:51.510 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:01.558 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:02.490 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:02.570 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:03.581 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:04.592 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:06.605 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:07.616 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.627 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:11.509 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:12.639 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:13.652 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:14.662 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:20.674 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:21.689 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:22.491 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:22.703 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:30.718 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:31.510 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:31.732 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:32.744 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:42.488 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:42.760 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:43.772 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:44.784 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:51.506 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:56.803 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:57.817 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:58.831 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:02.484 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:11.501 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:12.852 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:13.864 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:14.876 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:22.484 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:30.898 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:31.502 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:31.911 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.922 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:42.480 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:50.946 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:51.501 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:51.958 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:52.970 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:02.338 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:12.995 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:43:14.007 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:14.488 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:15.018 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:16.029 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:18.041 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:19.052 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:20.064 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:23.716 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:24.077 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:25.090 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.105 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.118 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:32.734 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:33.131 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:34.144 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:41.751 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:42.159 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:43.171 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:44.183 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:54.199 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:54.484 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:55.211 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:56.223 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:03.708 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:08.240 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:09.251 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:10.263 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:12.729 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:21.750 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:24.286 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:25.298 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:26.310 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:34.492 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:42.339 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.351 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:43.712 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:44.362 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:52.734 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:01.755 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:02.386 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:03.400 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.414 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:14.496 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:23.514 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:24.444 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:25.458 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:26.469 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:33.062 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:42.080 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:48.500 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:49.512 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:50.524 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:54.491 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:03.709 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:12.726 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:14.553 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:15.570 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:16.586 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:21.745 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:34.488 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:42.613 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:43.623 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:43.709 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:44.635 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:52.726 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:01.747 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:12.663 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:13.673 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:14.488 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:14.684 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:23.704 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:32.722 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:41.741 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:44.720 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:47:45.733 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:46.744 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.770 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:49.782 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:50.794 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:51.812 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:54.484 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:55.827 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:56.839 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.851 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:03.700 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:03.867 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:04.882 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:05.895 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:12.720 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:13.908 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:14.920 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.933 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:21.738 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:25.954 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:26.968 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:27.979 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:34.480 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:39.996 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:41.007 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:42.025 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:43.696 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:52.714 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:56.046 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:57.057 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:58.069 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:01.735 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:14.092 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:14.477 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:15.104 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:16.115 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:23.692 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:32.712 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:34.136 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.149 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:36.159 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:41.731 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:54.472 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:56.184 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:57.195 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:58.206 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:03.688 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:12.707 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:20.235 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.247 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:21.724 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:22.261 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:34.464 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:43.682 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:46.286 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:47.298 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:48.312 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:52.700 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:01.718 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:14.343 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:14.460 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:15.354 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:16.365 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:23.676 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:32.694 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:41.712 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:44.395 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:45.407 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:46.418 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:54.044 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:03.065 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:12.084 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:16.453 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:17.467 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:18.481 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:24.456 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:33.476 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:42.494 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:50.514 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:51.511 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:51.526 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:52.537 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:02.332 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:14.454 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:23.668 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:24.572 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:25.586 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:26.597 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:32.687 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:41.704 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:54.456 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:58.634 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:59.651 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:00.662 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:03.672 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:12.692 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:21.713 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:32.694 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:33.707 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:34.457 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:34.719 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:43.670 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:52.688 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:01.706 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:06.752 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:07.764 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:08.775 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:13.918 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:22.937 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:31.956 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:40.815 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:41.827 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:42.840 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:44.459 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:53.476 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:03.022 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:12.040 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:14.878 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:15.891 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:16.908 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:56:17.919 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.932 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:19.943 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:21.954 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:22.970 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:23.984 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:24.462 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:27.997 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:29.008 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:30.018 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:33.658 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:36.034 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:37.046 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:38.057 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:42.678 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:46.076 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:47.088 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:48.099 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:51.698 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:58.116 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:59.134 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:00.145 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:04.470 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:13.656 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:17.165 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:18.181 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:19.197 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:22.674 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:31.692 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:33.224 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:34.235 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:35.250 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:43.716 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:51.271 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:52.282 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:52.734 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:53.294 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:01.753 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:58:11.319 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:12.332 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:13.343 1 ERROR oslo.messaging._drivers.impl_rabbit [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] [502a6937-9f0a-44d1-993b-e2abb952f35b] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:14.465 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 99] Cannot assign requested address)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 99] Cannot assign requested address)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:21.462 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db [req-14509563-564c-47ba-9cac-adcb99aa6801 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 99] Cannot assign requested address)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 381, in save >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db self._check_minimum_version() >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 356, in _check_minimum_version >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db minver = self.get_minimum_version(self._context, self.binary) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 460, in get_minimum_version >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db use_slave=use_slave) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db result = fn(cls, context, *args, **kwargs) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 437, in get_minimum_version_multi >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db context, binaries, use_slave=use_slave) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 225, in wrapper >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 423, in _db_service_get_minimum_version >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return db.service_get_minimum_version(context, binaries) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 123, in service_get_minimum_version >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return IMPL.service_get_minimum_version(context, binary) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 270, in wrapped >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 456, in service_get_minimum_version >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return dict(min_versions) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 99] Cannot assign requested address)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:31.460 1 ERROR nova.servicegroup.drivers.db >2018-07-16 11:04:57.463 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -1 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:10.473 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -2 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:21.373 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -3 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:31.379 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -4 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:41.389 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -5 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:51.401 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -6 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:01.411 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -7 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:11.421 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -8 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:21.432 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -9 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:31.448 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:41.451 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -11 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:51.462 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -12 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:01.472 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -13 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:11.483 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -14 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:21.492 1 WARNING oslo_db.sqlalchemy.engines [req-4c837532-5741-4004-9a30-ef745b559aa2 - - - - -] SQL connection failed. -15 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines Traceback (most recent call last): >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 73, in _connect_ping_listener >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines connection.scalar(select([1])) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines return meth(self, multiparams, params) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines compiled_sql, distilled_params >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines context) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines cursor.execute(statement, parameters) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines result = self._query(query) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines conn.query(q) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines self._affected_rows = self._read_query_result(unbuffered=unbuffered) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines result.read() >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines first_packet = self.connection._read_packet() >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 998, in _read_packet >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines "Lost connection to MySQL server during query") >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:35.584 1 ERROR oslo_db.sqlalchemy.engines >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2125, in contextual_connect >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db **kwargs) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 110, in __init__ >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db self.dispatch.engine_connect(self, self.__branch) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__ >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db fn(*args, **kw) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 83, in _connect_ping_listener >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db connection.scalar(select([1])) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 880, in scalar >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return self.execute(object, *multiparams, **params).scalar() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return meth(self, multiparams, params) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return connection._execute_clauseelement(self, multiparams, params) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db compiled_sql, distilled_params >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1132, in _execute_context >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db None, None) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1125, in _execute_context >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db conn = self._revalidate_connection() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 429, in _revalidate_connection >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db self.__connection = self.engine.raw_connection(_connection=self) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2188, in raw_connection >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db self.pool.unique_connection, _connection) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2164, in _wrap_pool_connect >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db util.reraise(*sys.exc_info()) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 345, in unique_connection >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 931, in connect >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db self._get_server_information() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1245, in _get_server_information >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db packet = self._read_packet() >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:38.712 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:38:44.864 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds. Client port: None: error: [Errno 104] Connection reset by peer >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 931, in connect >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db self._get_server_information() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1245, in _get_server_information >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db packet = self._read_packet() >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 987, in _read_packet >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db packet_header = self._read_bytes(4) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1033, in _read_bytes >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db CR.CR_SERVER_LOST, "Lost connection to MySQL server during query") >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:38:45.589 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:38:45.881 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:46.897 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:47.909 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:49.923 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:50.952 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:51.967 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:55.979 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:56.990 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:38:58.002 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:04.015 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:39:05.026 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:05.586 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:06.037 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:07.048 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:09.060 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:10.071 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:11.083 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.095 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:15.596 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:16.106 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:17.118 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:23.137 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:24.148 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.160 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:25.600 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:33.172 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:34.183 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.194 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' (timed out)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:35.608 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:43.053 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:45.210 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:46.056 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:46.221 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:47.232 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:39:58.082 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:39:59.250 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:00.262 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:01.273 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:07.510 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:15.294 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:40:16.305 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:16.530 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:17.316 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:18.327 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:20.337 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:21.353 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:22.364 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:26.380 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:27.391 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.403 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:28.556 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:34.417 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:35.436 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.448 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:36.478 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:44.462 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:45.476 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:46.487 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:48.505 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:40:56.479 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:40:56.503 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:57.515 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:40:58.526 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:08.504 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:10.544 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:11.555 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:12.566 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:16.476 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:26.591 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:27.603 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:28.502 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:28.615 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:36.476 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:44.636 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:45.651 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:46.662 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:48.500 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:41:56.472 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:04.684 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:05.694 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:06.705 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:08.496 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:16.470 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:26.726 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:42:27.737 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:28.496 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:28.748 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:29.759 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:31.771 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:32.782 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:33.795 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:36.468 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:37.808 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:38.822 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:39.833 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:45.846 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:46.860 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:47.872 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:48.495 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:55.886 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:42:56.326 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:42:56.897 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:42:57.912 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:07.929 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:08.350 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:08.944 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:09.956 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:17.494 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:21.972 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:22.985 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:23.996 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:26.722 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:35.740 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:38.018 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:39.028 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:40.041 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:47.764 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:56.061 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:57.072 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:43:57.490 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:43:58.083 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:06.714 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:15.738 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:16.107 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:17.119 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:18.131 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:27.767 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:37.498 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:38.154 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:39.165 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:40.179 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:46.718 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:44:55.740 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:02.206 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:03.217 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:04.231 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:07.766 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:17.502 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:26.520 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:28.260 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:29.274 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:30.287 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:36.068 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:48.092 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:56.316 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.329 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:45:57.498 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:45:58.340 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:06.714 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:15.732 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:26.372 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:27.383 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:27.764 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:28.396 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:37.494 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:46.714 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:46:55.732 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:46:58.435 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:46:59.447 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:00.458 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:01.469 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:03.481 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:04.493 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:05.504 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:07.758 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:09.516 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:10.527 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:11.539 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:17.494 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:17.550 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:18.563 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:19.574 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:26.711 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:27.588 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:28.599 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:29.610 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:35.728 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:39.627 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:40.645 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:41.659 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:47.759 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:47:53.678 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:54.690 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:55.700 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:47:57.490 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:06.708 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:09.721 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:10.732 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:11.743 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:15.726 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:27.750 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:27.765 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:28.778 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:29.792 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:37.486 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:46.702 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:48:47.816 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:48.827 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:49.838 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:48:55.721 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:07.747 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:09.861 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:10.873 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:11.885 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:17.484 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:26.700 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:33.912 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:34.922 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:35.718 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:35.933 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:47.742 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:49:57.478 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:49:59.958 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:00.970 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:01.982 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 26 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:06.694 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:15.712 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:27.736 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:28.012 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:29.022 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:30.034 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 28 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:37.471 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:46.688 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:50:55.707 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:50:58.063 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:50:59.077 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:00.088 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 30 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:07.731 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:17.466 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:26.682 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:30.117 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:31.130 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:32.143 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:35.700 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:47.724 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:51:57.050 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:04.179 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:05.190 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:06.072 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:06.202 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:18.098 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:27.462 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:36.482 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:38.233 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:39.245 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:40.256 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:48.506 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:52:56.318 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:08.346 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:12.287 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:13.298 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:14.313 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:17.460 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:26.674 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:35.693 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:46.349 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:47.361 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:47.718 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:53:48.373 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:53:57.462 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:06.678 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:15.698 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:20.410 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:21.426 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:22.442 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:27.724 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:37.464 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:46.676 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:54.480 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:55.495 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:54:55.694 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:54:56.510 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 32 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:07.721 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:16.924 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:25.944 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:28.548 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:29.563 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:30.575 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: <AMQPError: unknown error>. Trying again in 1 seconds. Client port: None: RecoverableConnectionError: <AMQPError: unknown error> >2018-07-16 10:55:31.585 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:32.597 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:33.609 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 2 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:35.623 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:36.637 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:37.648 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 4 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:37.968 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:41.661 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:42.677 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:43.689 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:47.464 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:49.701 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:50.713 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:51.728 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 8 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:55:56.482 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:55:59.745 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:00.757 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:01.770 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 10 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:06.028 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:11.787 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:12.799 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:13.812 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 12 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:18.052 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:25.832 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:26.844 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:27.469 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:27.856 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 14 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:36.664 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:41.876 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:42.887 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:43.897 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 16 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:45.685 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:56:57.710 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:04.920 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: timed out. Trying again in 1 seconds. Client port: None: timeout: timed out >2018-07-16 10:57:05.933 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:06.949 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 18 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:07.476 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:16.663 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:25.681 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:25.975 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:26.986 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:27.998 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 20 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:37.704 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:46.723 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:57:48.021 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:49.033 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:50.045 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 22 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:57:55.740 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:07.770 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:58:12.070 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-1.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:13.082 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-2.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:14.094 1 ERROR oslo.messaging._drivers.impl_rabbit [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] [528db572-e7d0-4592-bae5-1d99857bdbb9] AMQP server on controller-0.internalapi.localdomain:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 24 seconds. Client port: None: error: [Errno 111] ECONNREFUSED >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:17.472 1 ERROR nova.servicegroup.drivers.db >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db [req-feb4fce0-1a90-4511-b885-8a8164c93f59 - - - - -] Unexpected error while reporting service status: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 99] Cannot assign requested address)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db Traceback (most recent call last): >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/servicegroup/drivers/db.py", line 91, in _report_state >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db service.service_ref.save() >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return fn(self, *args, **kwargs) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 382, in save >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db db_service = db.service_update(self._context, self.id, updates) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 189, in service_update >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 147, in wrapper >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db ectxt.value = e.inner_exc >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db self.force_reraise() >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db six.reraise(self.type_, self.value, self.tb) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/oslo_db/api.py", line 135, in wrapper >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return f(*args, **kwargs) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 240, in wrapped >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 569, in service_update >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db service_ref = service_get(context, service_id) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 255, in wrapped >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return f(context, *args, **kwargs) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 429, in service_get >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db result = query.first() >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db ret = list(self[0:1]) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__ >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return list(res) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__ >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return self._execute_and_instances(context) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db close_with_result=True) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db **kw >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db conn = self.session.connection(**kw) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db execution_options=execution_options) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1040, in _connection_for_bind >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db engine, execution_options) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 409, in _connection_for_bind >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db conn = bind.contextual_connect() >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db self._wrap_pool_connect(self.pool.connect, None), >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db e, dialect, self) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db util.raise_from_cause(newraise, exc_info) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db reraise(type(exception), exception, tb=exc_tb, cause=cause) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return fn() >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return _ConnectionFairy._checkout(self) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db fairy = _ConnectionRecord.checkout(pool) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 537, in checkout >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db rec.checkin() >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db compat.reraise(exc_type, exc_value, exc_tb) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 534, in checkout >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db dbapi_connection = rec.get_connection() >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 623, in get_connection >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db self.__connect() >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db connection = pool._invoke_creator(self) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return dialect.connect(*cargs, **cparams) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return self.dbapi.connect(*cargs, **cparams) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db return Connection(*args, **kwargs) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 706, in __init__ >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db self.connect() >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 963, in connect >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db raise exc >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 99] Cannot assign requested address)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 10:58:25.669 1 ERROR nova.servicegroup.drivers.db >2018-07-16 11:04:51.451 1 WARNING oslo_db.sqlalchemy.engines [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:04.460 1 WARNING oslo_db.sqlalchemy.engines [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] SQL connection failed. 9 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:14.466 1 WARNING oslo_db.sqlalchemy.engines [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] SQL connection failed. 8 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:25.370 1 WARNING oslo_db.sqlalchemy.engines [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] SQL connection failed. 7 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:35.381 1 WARNING oslo_db.sqlalchemy.engines [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] SQL connection failed. 6 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:45.396 1 WARNING oslo_db.sqlalchemy.engines [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] SQL connection failed. 5 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on '10.10.125.108' ([Errno 111] ECONNREFUSED)") (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:05:55.413 1 WARNING oslo_db.sqlalchemy.engines [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] SQL connection failed. 4 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:05.416 1 WARNING oslo_db.sqlalchemy.engines [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] SQL connection failed. 3 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:15.427 1 WARNING oslo_db.sqlalchemy.engines [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] SQL connection failed. 2 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:25.438 1 WARNING oslo_db.sqlalchemy.engines [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] SQL connection failed. 1 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:35.447 1 CRITICAL nova [req-2dc40483-b8f7-4c61-82c5-f179a7588b98 - - - - -] Unhandled error: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:35.447 1 ERROR nova Traceback (most recent call last): >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/bin/nova-scheduler", line 10, in <module> >2018-07-16 11:06:35.447 1 ERROR nova sys.exit(main()) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/nova/cmd/scheduler.py", line 47, in main >2018-07-16 11:06:35.447 1 ERROR nova topic=scheduler_rpcapi.RPC_TOPIC) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 258, in create >2018-07-16 11:06:35.447 1 ERROR nova periodic_interval_max=periodic_interval_max) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/nova/service.py", line 127, in __init__ >2018-07-16 11:06:35.447 1 ERROR nova self.manager = manager_class(host=self.host, *args, **kwargs) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 63, in __init__ >2018-07-16 11:06:35.447 1 ERROR nova invoke_on_load=True).driver >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 61, in __init__ >2018-07-16 11:06:35.447 1 ERROR nova warn_on_missing_entrypoint=warn_on_missing_entrypoint >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 81, in __init__ >2018-07-16 11:06:35.447 1 ERROR nova verify_requirements) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 203, in _load_plugins >2018-07-16 11:06:35.447 1 ERROR nova self._on_load_failure_callback(self, ep, err) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 195, in _load_plugins >2018-07-16 11:06:35.447 1 ERROR nova verify_requirements, >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 158, in _load_one_plugin >2018-07-16 11:06:35.447 1 ERROR nova verify_requirements, >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 227, in _load_one_plugin >2018-07-16 11:06:35.447 1 ERROR nova obj = plugin(*invoke_args, **invoke_kwds) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 43, in __init__ >2018-07-16 11:06:35.447 1 ERROR nova super(FilterScheduler, self).__init__(*args, **kwargs) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/nova/scheduler/driver.py", line 48, in __init__ >2018-07-16 11:06:35.447 1 ERROR nova invoke_on_load=True).driver >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/driver.py", line 61, in __init__ >2018-07-16 11:06:35.447 1 ERROR nova warn_on_missing_entrypoint=warn_on_missing_entrypoint >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 81, in __init__ >2018-07-16 11:06:35.447 1 ERROR nova verify_requirements) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 203, in _load_plugins >2018-07-16 11:06:35.447 1 ERROR nova self._on_load_failure_callback(self, ep, err) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 195, in _load_plugins >2018-07-16 11:06:35.447 1 ERROR nova verify_requirements, >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/named.py", line 158, in _load_one_plugin >2018-07-16 11:06:35.447 1 ERROR nova verify_requirements, >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/stevedore/extension.py", line 227, in _load_one_plugin >2018-07-16 11:06:35.447 1 ERROR nova obj = plugin(*invoke_args, **invoke_kwds) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 359, in __init__ >2018-07-16 11:06:35.447 1 ERROR nova self._init_aggregates() >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 372, in _init_aggregates >2018-07-16 11:06:35.447 1 ERROR nova aggs = objects.AggregateList.get_all(elevated) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper >2018-07-16 11:06:35.447 1 ERROR nova result = fn(cls, context, *args, **kwargs) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/nova/objects/aggregate.py", line 533, in get_all >2018-07-16 11:06:35.447 1 ERROR nova _get_all_from_db(context)] >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 985, in wrapper >2018-07-16 11:06:35.447 1 ERROR nova with self._transaction_scope(context): >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ >2018-07-16 11:06:35.447 1 ERROR nova return self.gen.next() >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1035, in _transaction_scope >2018-07-16 11:06:35.447 1 ERROR nova context=context) as resource: >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ >2018-07-16 11:06:35.447 1 ERROR nova return self.gen.next() >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 638, in _session >2018-07-16 11:06:35.447 1 ERROR nova bind=self.connection, mode=self.mode) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 403, in _create_session >2018-07-16 11:06:35.447 1 ERROR nova self._start() >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 489, in _start >2018-07-16 11:06:35.447 1 ERROR nova engine_args, maker_args) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 513, in _setup_for_connection >2018-07-16 11:06:35.447 1 ERROR nova sql_connection=sql_connection, **engine_kwargs) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator >2018-07-16 11:06:35.447 1 ERROR nova return wrapped(*args, **kwargs) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 184, in create_engine >2018-07-16 11:06:35.447 1 ERROR nova test_conn = _test_connection(engine, max_retries, retry_interval) >2018-07-16 11:06:35.447 1 ERROR nova File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 370, in _test_connection >2018-07-16 11:06:35.447 1 ERROR nova six.reraise(type(de_ref), de_ref) >2018-07-16 11:06:35.447 1 ERROR nova File "<string>", line 2, in reraise >2018-07-16 11:06:35.447 1 ERROR nova DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:35.447 1 ERROR nova >2018-07-16 11:06:39.506 1 WARNING oslo_db.sqlalchemy.engines [req-934a2380-ef90-4c63-9282-8e23335f572f - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:49.518 1 WARNING oslo_db.sqlalchemy.engines [req-934a2380-ef90-4c63-9282-8e23335f572f - - - - -] SQL connection failed. 9 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:06:59.529 1 WARNING oslo_db.sqlalchemy.engines [req-934a2380-ef90-4c63-9282-8e23335f572f - - - - -] SQL connection failed. 8 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:09.540 1 WARNING oslo_db.sqlalchemy.engines [req-934a2380-ef90-4c63-9282-8e23335f572f - - - - -] SQL connection failed. 7 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8) >2018-07-16 11:07:19.549 1 WARNING oslo_db.sqlalchemy.engines [req-934a2380-ef90-4c63-9282-8e23335f572f - - - - -] SQL connection failed. 6 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=1459172">View the attachment on a separate page</a>.</b>
View Attachment As Raw
Actions:
View
Attachments on
bug 1601472
:
1459162
|
1459171
| 1459172